What Are Crawl Errors And How To Remove Them Using GWT?
October 5th, 2014 | by Ravi Chahar || 50 Comments |
While handling your website you may get some errors. You can check about crawl errors by visiting Google Webmaster Tools (GWT). You all know that Google bots crawl your website and if they face any problem in crawling then GWT is the one which will let you know about those crawl errors.
Many bloggers ask for proper guidance regarding errors but do they all know what are crawl errors?
Let me tell you. When Google tries to crawl your website and doesn’t able to crawl any particular page or your website then these errors are termed as crawl errors. As its name is signalling us about its meaning related to crawling.
How to fix crawl errors in WordPress?
After getting any error many bloggers get afraid, but there is nothing to panic about. Though there are mainly two types of crawl errors categorized as:-
1). Site Errors
2). URL errors
Site Errors :- Let me tell you about both types of errors. When Google bots crawl your whole website and find an error related to the server, domain name or robots.txt file then these type of errors are considered as site errors. Google Webmaster Tools shows you the data of crawling of last 90 days and you can check all the errors there.
-> DNS error :- As you, all know that DNS stands for Domain Name System. This error is shown by GWT when Google is not able to connect your DNS server which is the company from which you have bought the domain name of your website. In most of the cases, your DNS and web hosting is same because most of the bloggers get one free domain name with their web hosting.
You may get this error after moving your website from one web host to another one. Though Google stats that main cause of getting DNS error is the lost connection from DNS server. If you want to fix this error then you must contact to your DNS provider to check the connectivity for your domain name.
If your website is getting this error then you must try to fetch the URL of your website using “fetch as Google”. Though this error is not a problem for your website because it doesn’t stop Google to crawl your website but it may create a problem for your readers while visiting your website.
-> Server connectivity errors :- These errors may be shown to you because Google is facing problem in connecting the web host of your website. The server may be down or may be too slow for which Google drop the request and shows server connectivity errors. There are many messages you may see during this period.
When you try to open your website then you may get connection timeout, connection failed, truncated response and many error messages like these. These errors are shown just because of the web hosting service of your website. You may notice that when any web hosting company faces downtime then at that time you can’t access the website which is placed on their server.
You must know how to fix crawl errors in WordPress? And server connectivity errors are major to panic about because while getting these errors no one can visit your website because of server failure.
You can fix these errors by contacting your web hosting provider to provide you quality service. You must take care that home page size of your website shouldn’t be high because it can cause a problem while crawling. Lower the page size, great will be the speed of your website.
Many people stop Google to crawl their website. I am not talking about all the Google bots, there are few to be ignored. Choose your web hosting provider wisely.
-> Robots.txt fetch errors :- You all are well aware of the robots.txt file which tells Google bots what to crawl and what not to crawl in your website? When Google bots start crawling any website, the first thing to see the robots.txt file which guides then about crawling of the website. When Google bots don’t get this file then they return back and wait for some time according to their crawling schedule and try to crawl your website again.
When they again don’t get the robots.txt file to tell about crawling for your website then Google Webmaster Tools shows you this error. You all must know that robots.txt file is the only reason for getting this error. You all must check this file in regular interval of time.
If you don’t want any particular folder to be crawled by Google bots then add it in your robots.txt file. If you want to crawl everything in your website then you don’t need to have this file. Though Google bots adds 404 error but still they will continue to crawl your website.
If you are using any firewall to control crawling for your website then you must keep in mind that it shouldn’t be adjusted to Google blocking.
URL Errors :- These errors are related to particular pages of Desktop, Smartphone, Featured phone and Android phone. Google uses different mechanism while crawling your website for desktop, smartphones and for many other devices. You may have read about choosing a good theme for your website which is mobile friendly. Its the main reason because Google bots crawl your website for mobile phones too.
If they get any kind of difficulty while crawling pages of your website in reference to all the devices then GWT shows URL errors. There may be any certain page which is not good for any particular device to show then GWT shows an error for that page.
If you want to fix URL errors then you must always redirect with 301 redirects. Maintain your sitemap. If you get any URL error then after going through it, you should mark them as fixed in Google Webmaster Tools.
Keep in your mind that deep links for any device should be maintained properly so that Google bots can crawl them easily.
Now it’s your turn to fix Google errors :-
Do you know what are crawl errors? How to fix crawl errors in WordPress? How to fix Google crawl errors using GWT?
There are many beginners who used to ask these questions because they need to know about Google crawling and all the errors. After going through this post you may get an idea about all the errors show by Google Webmaster Tools.
Related Posts
50 comments
Leave a Reply
Hi Ravi,
Once again amazing post. As I have mentioned before that you are the master of WordPress. Again you are here with another new tech. Crawl errors are really very common through which some times our pro bloggers always suffer so, every blogger must have awareness about all these errors and how to fix these errors.
This post is really very helpful specially for me as my own field is blogging in WordPress. Hope others will also get benefit from it.
Thanks for sharing such an excellent post.
Hi Mairaj,
Thanks for appreciation.:)
I am pleased to know that you consider me good in WordPress.
Many bloggers get crawl errors for their website but they hardly know about them. They should know about these errors so that they can fix them and crawling of their website will become easy.
Thanks for taking time to say your words.
Have a nice week ahead.:)
~Ravi
Hi Ravi,
Informative post, yet again ?
It reminds me of the time, not long back when I was so new to such errors. But when I read about it all on another blogger’s blog and more of it on Google, I understood it all. Yes, Crawl errors are clearly visible in Webmaster Tools and you can immediately see and make amends, depending on the various kind of errors, just as you mentioned.
I’ve got such errors sometimes also, and knowing how to fix such errors is the important part – glad you explained how to do that with each of the errors too. I know this post, like your others, is going to help many bloggers, new ones and even seasoned ones, because not everyone is aware of them, especially the power of GWT and how all it can help you. All you need to do is keep a regular check on it, isn’t it?
Thanks for sharing. Have a nice week ahead ?
Hi Harleena,
After visiting to Google Webmaster Tools crawl errors are the first thing you will see.
But there are many people who don’t even know what are these crawl errors? Though they see these errors in GWT for their website but don’t have any idea as why they get errors? How to fix them?
Its necessary to know about such type of errors to maintain your website properly.
Yeah! you must check them regularly.
Glad to hear from you.:)
Have an amazing day.
~Ravi
thank you for explaining the crawl errors in detail. it’s truly usefulto learn why the errors occur and what do they mean when you have them in your Google Webmaster Tools. However I was expecting more on how to solve these errors in WordPress. you have nicely explained these error messages and for many cases we need to contact our hosting providers. I was interested to learn what else can we do in WordPress to solve any of these problems? Expecting more of it to see in your coming posts. Thank you once again.
Hey Suresh,
Though there are many crawl errors about which you should know but still in my opinion server connectivity and robots.txt error are the worst.
I am glad to hear that you are interested in learning more about these errors and their solution. I agree with you that in most of case you need to contact to your web hosting providers. But for solving robots.txt error you need to know about this simple file.
You can edit it your own as I have explained about it in my previous post. Robots.txt errors can be solved by editing it properly.
Thanks for taking time to drop your amazing input.
Have a nice week ahead.:)
~Ravi
Hi Ravi
I wish I had seen this when I first started, my learning process could have been much easier. A very informative post with great illustrations and cool ways to fix the errors.
Hi Ikechi,
We always wish to have some easy guide to learn more but in our starting days we don’t get it.
I hope through these type of articles beginners will come to know about some simple but necessary things related to website building.
Glad to see you around.:)
Enjoy your week ahead.
~Ravi
Hey Ravi,
Great explanation. People often panic when they show me their webmaster tools and there’s a bunch of errors OR people usually panic when they see their impressions drop a lot, then I just pinpoint it because they had soooo many crawl errors.
It’s amazing what this simple thing can do. Good piece Ravi
Hey Dennis,
These are some simple things about which you should know.
If you don’t know about these crawling errors then it would not by possible for you to solve them.
Google may stop crawling your website which means your website will not have any existence on the World Wide Web.
Thanks for taking time to share your thoughts.
Hope you are enjoying your stay.:)
~Ravi
Hi Ravi,
Google Webmaster Tools is a great way to detect errors and to detect the search queries as well.
A very helpful post indeed, loved that way you explained, thanks.
Hi Vishnu,
I am glad that you liked this post.:)
Knowing about crawling errors is necessary and bloggers should know how to fix them.
GWT is an amazing tool which will let you know about all the errors.
Thanks for visiting.
Have a great day.:)
~Ravi
Hey Ravi,
Really it was indeed a great post, when i was new to blogging i had seen robot.txt error but not given much importance to it as i was not aware of it but slowly slowly i get to know about this and this was the main reason i was not getting organic traffic.
This post will surely help others to know about different types of crawl errors
Thanks for sharing
Regards
Suprabhat
Hi Suprabhat,
In the beginning of every business we don’t know about everything. Its just we need to learn things slowly.
When time passes we come to know about more stuffs.
While blogging you should know about these errors so that you can have perfect crawling for your website.
Thanks for stopping by.
Enjoy your week.:)
~Ravi
Hello Ravi,
Yet again a informative and detailed post. Its great to read these tips and so I have bookmarked this.
I at first did check GWT but not as detailed as you show. Just a fetch as google and some modifications here and there.
Thanks for bringing out a great topic.
Hi Swadhin,
GWT shows crawl errors on the dashboard and you can easily see the status just by clicking at crawl errors.
Then you will see two type of errors as I have mentioned in this post.
Its necessary to keep an eye at them.
Thanks for your regular support.:)
Have a great weekend.:)
~Ravi
Thanks for taking the time to explain crawl errors clearly. A lot of people are unfamiliar with them, and overreact when GWT shows they have crawl errors. Most of the time, removed pages and failure to integrate 301 redirects are the root causes.
Thanks for sharing!
Hey James,
Glad to see you after long time.:)
Its necessary to do 301 redirect so that Google bots will not show crawl error for any page which has removed.
You should know about all the crawl errors so that you can solve them properly.
Thanks for taking time to say your words.
Enjoy your stay.:)
~Ravi
Hi Ravi,
I know many people who get panicked after seeing their crawl errors irrespective of the fact that they can be handled easily and fixed with just a click. You have explained it really well. But sometimes webmaster gives conflicting and old results which update later even after you have resolved the issue. Can you explain about the search queries, CTR and Impressions a website get and how the position in GWT is different from the exact rankings in SERPs.?
Hi Aditya,
You know no one can guess the real ranking ranking of your website. You may be well known about Alexa ranking and Google ranking. There is nothing to worry about them The only thing is thinking about your readers. Write for them not for getting higher SERPs.
GWT shows the results based on crawling done by Google bots. There are millions of Google bots which crawl millions of websites everyday. When you solve crawl errors then its not possible to get them solved in GWT because it will not show until Google bots crawl your website again.
CTR is based on the number of times your ads were shown and number of times you got click at your ads. Google has its own thing to do with it. In my opinion stick to your content and drive traffic to your blog and CTR will be increased.
Thanks for putting your wonderful input.
Have fun this weekend.:)
~Ravi
Hello Ravi,
Very informative post indeed. This time, I can say it’s helpful for me. Because, I haven’t cleared my crawl errors due to less time. I will definitely look into these steps and solve my all crawl errors as soon as possible.
Thanks for reminding and sharing this beautiful tutorial. Have a good day!
Hi Nisha,
Knowing about crawl errors is very important from SEO point of view. I mean if Google is not able to crawl your website then it signifies that your website doesn’t exist.
Its good to check about such errors to be safe.
Thanks for taking time to say your words.
Enjoy your day.:)
~Ravi
Hi Ravi,
WOW!! Very informative post and well explained.
I am one of those who don’t know and have no clue about crawl errors. Thanks for getting my attention on this one as I will bookmark this post and go to GWT and look if there I have any errors and follow your suggestions here to solve them.
Thanks Ravi for taking the time to write such a useful post. I will definitely follow these steps and solve all the crawl errors that I may have.
Have a great week ahead.
Be Blessed,
Neamat
Hi Neamat,
You know when I have started my blogging career I was not aware about such type of things. I have faced many problems due to crawl errors as it was hard to learn about them.
GWT is an amazing tool to use. You can know that status of your website using it.
Google has provide this free tool to help bloggers so that they can handle all the errors.
I am glad to know that it will help you.
Thanks for your wonderful input.
Have a successful week ahead.:)
~Ravi
Hey Ravi,
Really it was indeed a great Article Good Job my dear , i like Your Content , Post Thanks For Explen , & Sharing me , Thank You Vey very Much , Like It
Regard
Leena Shah ,
Hi Leena,
You must know that sharing is caring for which we are here in this online community.
Making connections is important and human bonds last longer.
Thanks for appreciating.
Hope to see you soon.:)
~Ravi
Hi Ravi,
These is a very brilliant post bro. I was actually facing this same issue after migrating from Blogger to WordPress, but in few days I was able to figure out and immediately fixed the error. I believe these post will be of a help to mostly the newbie bloggers who are facing the same problem. Anyways, thanks for taking the time to write such a wonderful post. More greeze to your fingers. ?
Hey Victor,
We all face such type of problems while handling our website.
Its good to know about such errors. Using GWT you can see the status for your website.
I am glad that you have enjoyed this post.
Thanks for sharing your thoughts.
Have a nice day.:)
~Ravi
Hi Ravi
Technically a very useful post and one can understand the points without any problem even if he is non-techie.
Mostly bloggers visit the GWT just to check the number of impressions, links to site and other things which GA does not tell. But many fail to note what type of errors it is showing and which can be corrected without much effort.
Such errors bloggers mostly face after shifting from one blogging platform to another usually from Blogger to WordPress.
You in this post not only pointed all the main types of crawl errors but also put their solutions to make it a comprehensive post on the topic.
Thanks for sharing.
Hi Muba,
You may have met many non-techie bloggers who don’t put much attention to techie stuffs. But knowing about crawl errors is a must have knowledge.
Checking impressions and clicks is not enough. Bloggers should know about many others things using GWT.
You are right. When bloggers move their website from one platform to another then crawling errors are common. But its better to check them regularly so that your website doesn’t face any problem regarding crawling.
Thanks for your valuable comment.
Have an awesome week ahead.:)
~Ravi
Hello Ravi,
Very Informative post :). As Suprabhat told before even I have not paid attention to crawl error but I agree with you that according to time we learn a lot ?
Regards,
Minakshi Srivastava
Hey Minakshi,
Thanks for stopping by and dropping your words.
Have a great day.:)
~Ravi
Hey Ravi,
Your tutorials are really great and you sure do explain things well.
I remember learning all about this years ago and I spent an entire hour teaching my mastermind group about these. None of those ladies had ever even been to their webmaster tools so it was a little over their heads but so worth the time.
This is something that I’m sure to check at least once a month because there is always something.
I have a question for you and maybe you know the answer to this. Why will Google pick up a post URL that doesn’t exist. For instance, let’s say that my URL is myblog.net/posttitle/somethingreallyodd. It’s like it picks up another link from out of the blue and adds it to the end of mine and no matter how many times I delete it it keeps coming back. It was never a post title or URL period so how does that even happen? I even have them showing up as broken links. It drives me nuts.
Thanks Ravi for this great information and I sure hope you have a resolution for me. ?
Enjoy your week.
~Adrienne
Hi Adrienne,
I agree with you that most of bloggers don’t put an eye at these crawl errors. Though its visible to everyone when log in to Google Webmaster Tools but still bloggers don’t give much attention to them.
Due to busy schedule its hard to check about such errors once a month. I do it once in a week.
You know Adrienne, the same problem you are asking has faced by me last month. I used to get an error of a URL which didn’t even exist. I was really fed up of all this.
These kind of errors are shown due to some reasons. It may be due to bad host set up while providing 404 page. When 302 redirect is done then in most of cases bloggers do it to the home page which sometime create issues.
In some cases it is due to bad script which accepts any old parameter(it may be fake). You can check any such kind of problem related to scripts, style sheets of CSS, broken links by using “Xenu Link Slueth”. Its a website trusted by Google to help its users.
I hope you will get some idea about the real fact.
Thanks for sharing your extraordinary thoughts.:)
Have an awesome week ahead.
~Ravi
I have some old 404 not found errors, after Google crawl visit entire sitemap upon my request most errors dissapeared, only few still show up.
Great tutorial, I love what you share and I’ll subscribe.
Kind regards,
I. C. Daniel
Hi Daniel,
Welcome to my blog.:)
You may get many errors if you check in GWT. For removing 404 error you can redirect those posts or pages of your website to another so that Google bots won’t show any crawl error.
Though 404 error doesn’t affect your website but still you should fix it.
Glad to hear from you.
Have a wonderful day.:)
~Ravi
Hello Ravi, After reading your post and then the comments it does look like I may be the only one here that doesn’t know what these crawl errors were.
Looks like now I do though LOL and I know how to go about fixing them. I am going to head on over to Google Webmaster Tools to see if I can detect these crawl errors?
I had no idea that it was necessary to know about these crawl errors to maintain my website properly.
Thanks for sharing.. Chery :))
Hi Chery,
There is nothing much to know about crawl errors. Though its very important to solve these kind of errors because if you get any error then Google will not be able to crawl your website and your posts will not get indexed and even for Google your website will not exist on this World Wide Web.
Its good to know that after reading this article you have gotten some idea about crawl errors.
For your website safety you should know about these stuffs.
Thanks for stopping by and taking time to share your thoughts.
Have an amazing week ahead.:)
~Ravi
Hi Ravi,
Thanks for this informative article on Crawling errors. Sometimes it really becomes an headache.
Thanks for this article..
Hi Jeeshan,
First of all welcome to my blog.:)
Its good to hear that the article is worthy to read for you.
You should know about crawl errors in detail.
Thanks for stopping by.
Hope to see you soon.:)
~Ravi
Hey Ravi,
Most of the bloggers are really unfamiliar with such kind of stuff. You are doing an amazing job by putting these stuff together and posting on your blog.
You always come up with Informative yet amazing content. And this one was another piece of your informative writing.
Thanks again and do have a great week ahead!
Cheers!
– Rohit
Hi Rohit,
I have also noticed that there are many bloggers who visit GWT but don’t know about crawl errors.
It because most of bloggers don’t handle techie stuff of their blog own.
I am glad that you liked this article. It will help you to understand about crawl errors deeply.
Thanks for taking time to say your words.
You too have fun this week.:)
~Ravi
Hello Ravi,
Thanks for this wonderful post!!
I have being working on clearing these errors from long time but was not able to find the exact solution. I hope your post help will me to get rid of this headache now..
Dev
Hey Dev,
Welcome to my blog.:)
Most of error are due to web hosting. But robots.txt is because of you do something wrong in its configuration.
You should check about these errors once in a week so that you don’t get any issues regarding crawling of your website.
Thanks for stopping by.
Hope to see you again.;)
~Ravi
Hi Ravi,
Thanks for letting me know about the errors finder tools which is very essential for the bloggers like us.
I will give it a try as you introduce nice stuff with us.
Stay blessed. keep posting informative post like this.
Hi Shiwangi,
Good to see you after long time.:)
You should check about all the crawl errors by using Google Webmaster Tools.
Sometimes it may cause problem and your visitors will not be able to access your website.
Thanks for stopping by.
Have a wonderful day.:)
~Ravi
Thanks for this wonderful article Ravi… i was struggling to get my pages indexed in ggole as my crawler rate was high but now i think i can improve on that fro sure.. keep up the good work… Cheers
Hey Rajni,
Welcome to my website.:)
For sure you should check crawl errors for your website so that you can have better website.
Crawling and indexing are two different things which you need to learn.
Thanks for stopping by.
Enjoy your day.:)
~Ravi
Hey Ravi,
This was a great lesson on crawl errors. I actually check them every 4 to 5 months to see what errors I’m getting.
I had installed broken link checker to avoid those 404 errors. I’ve been there, done that and don’t want to hassle with it again. I had to correct a bunch of link, especially when I changed the structure of my permalink. I used the 301 redirect plugin and I was able to redirect about 80% of my permalinks within 5min.
Thanks for sharing Ravi! Have a great rest of the week!
Hi Sherman,
We all hate to see errors at our websites. Bloggers should know how to check and repair these errors. When we talk about broken links then it’s disastrous to seek them. It’s the good step towards security when you install a broken link checker.
Though I usually check my broken links at broken link checking websites but still plugins can do much better.
Permalinks structure is also which should be taken into consideration.
Glad to have you here.
Have a great weekend.
~Ravi